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Summary:  Battlespace  Management  systems  are  often 
developed  by  decomposing  the  problem  into  separate 
functions.  For  example,  the  battle  scene  is  decomposed 
into  intelligence  reports,  sensor  displays  for  each 
sensor,  contact  tracks  for  each  sensor  contact, 
environmental  (weather,  oceanography)  conditions  and 
predictions,  sensor  effectiveness  predictions, 
geophysical  / physical  oceanographic  pictures,  etc. 
Once  the  problem  has  been  decomposed  and  analyzed, 
the  decision  maker  must  put  it  back  together  in  a 
mental  information  fusion  process,  integrating 
information.  The  tools  to  help  the  expert  decision 
maker  re-fuse  the  problem  are  far  fewer  and  more 
difficult  to  develop  than  the  tools  to  decompose.  The 
research  reported  here  takes  an  alternative  approach  by 
providing  information  displays  that  cluster  and 
integrate  information  according  to  the  expert  decision 
maker's  knowledge  schema  and  procedural  structure.  A 
complex,  time-dependant  (but  non-military)  test 
domain  with  multiple,  conflicting  goals  was  selected. 
Functional  partitioning  required  greater  effort  while 
procedurally  based  information-clustering  resulted  in 
more  efficient  (timely  and  accurate)  decision  making. 

INTRODUCTION 

The  Problem:  Battlespace  Management  systems  are 
often  developed  by  partitioning  the  problem  into  many 
functions  or  tasks.  For  example,  one  tool  displays  all 
the  broadband  noise,  sensed  in  all  directions.  Another 
provides  a similar  display  of  narrowband  noise 
received  by  each  sensor.  Another  tool  makes 
environmental  (atmospheric  or  oceanographic) 
predictions  for  specific  variables  (wind  / current  speed, 
temperature  profile,  etc.).  Yet  another  tool  gives  the 
current  conditions.  Several  algorithms  work  to  solve 
the  target-motion-analysis  problem.  Thus  each  unit  in 
the  battlespace  is  decomposed  into  many  separate 
signals  and  kinematic  components,  all  independent 
from  one  another  and  from  their  underlying  physical 
location  and  the  constraints  that  it  imposes. 
Decomposition  facilitates  efficient  engineering  of  the 
algorithms  and  programs.  Each  function  has  its  own  set 
of  developers,  and  therefore,  its  own  set  of  tools.  The 
consequence  of  this  approach  is  that  the  problem  is 


split  into  many  different  unassociated  bits  of 
information.  However,  the  partitioning  scheme  is  not 
necessarily  congruent  with  the  way  that  the  decision 
maker  solves  the  problems. 

The  task  of  the  decision  maker  is  to  evaluate  the 
available  information,  predict  the  effects  of  various 
action  options,  and  communicate  the  decision. 
Evaluation  includes  integrating  data  from  the  different 
sources  described  above,  comparing  conditions  to 
assumptions,  assessing  accuracy,  etc.  The  available 
information  is  composed  partially  of  the  output  of 
various  tools  and  partially  of  unanalyzed  or  “raw”  data. 
It  includes  history,  current  state,  and  predictions.  It 
may  contain  considerable  uncertainty  and/or  may 
change  over  time.  Required  decisions  may  include  both 
what  to  do  and  when  to  act.  Information  management 
tools  and  decision  aids  are  developed  for  these  tasks 
because  they  are  so  complex  and  because  the 
supporting  information  is  so  complex  and  uncertain. 
However,  once  the  problem  has  been  decomposed  and 
analyzed,  the  decision  maker  must  put  it  back  together 
in  a mental  information-fusion  process,  integrating 
information  from  these  many  tools  and  phases.  The 
tools  to  help  the  expert  decision  maker  to  put  the 
problem  back  together  are  far  fewer  and  more  difficult 
to  develop  than  the  tools  to  take  it  apart. 

A Proposed  Solution:  An  alternative  approach  to 
decision  aid  development  is  to  start  with  an 
understanding  of  the  knowledge  and  procedures  of  the 
expert  decision  maker  and  then  design  tools  to  support 
these.  With  this  knowledge,  we  could  design 
information  management  decision  aids  in  the  way  that 
new  walkways  are  sometimes  planned.  That  is,  where 
natural  paths  occur  because  of  repeated  use  by 
pedestrians,  constructed  (concrete,  macadam,  gravel, 
etc.)  pathways  are  built.  In  the  same  way,  battlespace 
management  tools  and  decision  aids  should  provide 
support  for  knowledge  in  the  head  — the  procedural 
paths  we  create  through  the  task  and  the  information. 

Experts  make  use  of  the  procedural  components  of 
their  knowledge,  as  well  as  the  declarative  content: 
That  is,  they  know  how-to  as  well  as  what,  why,  when, 
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and  where.  Tools  used  to  do  a task  and  procedural 
knowledge  of  that  task  are  not  independent  entities. 
The  tool  can  facilitate  the  task  procedures  or,  in  the 
case  of  clumsy  automation  (Wiener,  1989),  dictate 
conflicting  procedures.  The  organization  of 
information  can  provide  the  cognitive  equivalent  of 
affordances  (Norman,  1988)  or  “handles”  that  facilitate 
performance  or  obstacles  that  hinder  it.  For  example, 
calculators  can  use  either  arithmetic  notation  or 
“reverse  Polish”  notation.  Arithmetic  notation  allows 
the  user  to  enter  numbers  and  operators  as  they  would 
on  paper  (3  + 2 =)  and  supports  the  average  user’s 
procedural  knowledge.  Reverse  notation  requires  the 
user  to  enter  numbers  first  and  then  operators  (3  2 + =). 
Although  this  notation  groups  like  information 
(numerals,  operators)  together,  it  requires  the  average 
user  to  reorder  the  information  from  normal  arithmetic 
procedures. 

There  is  a well  documented  interaction  between 
knowledge  in  the  head  and  information  in  the  world. 
Kleinmuntz  and  Schkade  (1993)  reviewed  several 
studies  that  show  how  problem  representation  affects 
the  speed  and  accuracy  of  identifying  and  assessing  the 
situation,  and  consequently,  the  quality  of  the  decisions 
made  with  that  information.  For  example,  Johnson, 
Payne  and  Bettman  (1988)  found  that  display  format 
effects  the  likelihood  of  preference  reversals  (a  well- 
documented  decision  error)  in  choice  decision  making. 
Decision  makers  in  these  studies  shifted  information 
gathering  strategies  as  a function  of  display  format. 
Brown  and  Klayman  (1989)  and  Smith  (1989)  found 
that  representation  affects  subjects'  ability  to  identify 
key  problem  elements  in  naturalistic  decision 
situations.  Larkin  (1989)  has  called  this  effect  display- 
based  problem  solving  because  the  availability  and 
form  of  the  information  displayed  can  affect  problem 
solving.  For  example,  Russo  (1977)  found  that  a table 
of  unit  prices  for  an  entire  category  of  food  facilitated 
price  comparison  and  decision  making  as  compared  to 
unit  prices  displayed  with  each  item,  although  unit 
prices  are  calculated  by  item,  not  category.  One  reason 
for  this  improvement  may  be  the  reduction  in  working 
memory  load  when  appropriate  information  is 
clustered.  Thus,  the  tools  used  to  do  a task  and 
procedural  knowledge  of  how  to  do  that  task  are  not 
independent  entities.  The  tool  can  facilitate  the  task 
procedures  or,  in  the  case  of  clumsy  automation 
(Wiener,  1989),  dictate  conflicting  procedures. 

To  solve  the  problem  posed  above  I propose  providing 
information  displays  that  cluster  and  integrate 
information  according  to  the  expert  decision  maker's 
knowledge  schema  and  procedural  structure  rather  than 
according  to  a functional  one.  I hypothesize  that  such  a 
system  would  lead  to  more  efficient  decision 
performance.  What  I mean  by  efficient  is  equal  or 
better  performance  in  a shorter  time,  with  less  effort. 


In  the  remainder  of  this  chapter  I will  first  report  on  an 
experiment  that  demonstrates  the  performance 
advantages  for  this  idea  and  then  I will  discuss  several 
military  applications  for  the  findings. 

TESTING  THE  HYPOTHESIS 

To  test  the  hypothesis  a complex,  time-dependant  (but 
non-military)  test  domain  with  multiple,  conflicting 
goals  was  selected.  The  decision  task  was  designed  to 
have  a one-to-one  correspondence  with  key  elements 
of  the  submarine  problem.  (One  advantage  of  the  non- 
military task  was  that  many  more  experts  were 
available.  Additional  evaluation  verified  the  task 
validity.)  Three  information  format  schemes, 
alphabetical  listing  (format  A),  functional  partitioning 
(format  B),  and  procedurally  based  information- 
clustering  (format  C)  were  tested.  Version  C was 
designed  by  a bootstrapping  procedure  based  on  two 
individuals  pilot  testing  versions  A and  B. 

Three  classes  of  dependent  measures  were  used.  The 
first  was  total  time-on-task.  The  second  reflected 
outcome  performance,  and  the  third  measured 
processing  activity.  Results  showed  that  version  C lead 
to  the  most  efficient  performance.  There  was  an 
interaction  between  performance  measures  and 
measures  of  processing  time  and  processing  effort. 
Functional  partitioning  required  greater  effort  for 
limited  performance  improvement  over  the 
alphabetical  format.  Thus,  the  right  organization 
scheme  can  provide  the  support  for  improved  cognitive 
performance. 

There  are  many  possible  partitioning  schemes  for 
categorizing  and  organizing  information.  Like  the 
contents  of  a computer  directory,  tools  on  a 
workbench,  or  merchandise  in  a store,  information  in  a 
decision  support  system  can  be  organized  by  many 
attributes,  including  size,  purpose,  time,  or  order  of 
use.  The  different  organizational  structures  facilitate 
achieving  different  goals.  Random  placement  speeds 
cleaning-up  after  a project,  but  organization  by  purpose 
speeds  retrieval  of  tools  from  a workbench. 

Phase  1:  Information  Organization:  The 

experimental  task  was  simulated,  on-line,  college 
course  scheduling.  This  task  has  many  elements  in 
common  with  the  target  task,  dynamic  decision  making 
under  uncertainty,  but  has  many  more  experienced 
individuals  to  serve  as  testers.  To  simulate  an  event- 
driven  environment,  classes  could  fill  while  the 
"student"  was  selecting  a schedule.  When  a planned 
course  was  filled,  the  subject  had  to  reassess  the 
situation  and  find  a new  course  that  fulfilled  the  other 
requirements.  Elements  of  data  history  were  important 
because  previous  semester's  records,  program 
requirements,  and  course  prerequisites  had  to  be 
reconciled.  A set  of  sometimes  conflicting  goals  further 
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constrained  choices  (see  Table  1).  Lastly,  of  course, 
classes  could  not  conflict  with  one  another.  To 
simulate  the  multiple  sources  of  information,  each 
function  (e  g.,  instructor  rating  lists,  course  schedules, 
requirements  lists,  student  history,  course  locations  and 
distance  maps,  etc.)  had  its  own  information 
presentation.  This  task  is  not  a simple  scheduling 
problem  because  there  is  no  single  “optimal”  or 
algorithmic  schedule  that  solves  all  constraints  and 
meets  all  the  goals.  It  requires  goal-driven  decision 
making  to  achieve  acceptable  performance. 

A scoring  scheme  was  developed  that  operationalized 
each  of  these  elements  as  values  associated  with 
accomplishing  the  goals  and  values  for  each  of  the 
choices  (i.e.,  courses  and  instructors).  It  was  predicted 
that  scores  would  reflect  the  level  of  organization  in 
the  display  schema. 

Table  1 : Goals,  listed  in  priority  order 

■ Register  for  15-17  hours  (5  courses). 

■ Try  to  fulfill  requirements  and  prerequisites  for 
both  general  education  and  your  major.  (Note 
that  you  may  remain  an  industrial  engineering 
major  or  select  engineering  psychology.) 

* Try  to  schedule  so  that  you  have  one  full  day 
or  two  half-days  off.  (One  full  day  is 
preferable.  Assume  you  have  a job  and  will 
otherwise  have  to  work  on  the  weekend.) 

■ Try  to  get  the  best  instructors  possible.  A list  is 
provided  of  instructor  ratings  from  a student 
survey. 

■ Try  to  avoid  8 am  classes. 


Information  in  each  of  the  sources  was  structured 
differently. 

Although  the  majority  of  these  materials  were 
organized  by  academic  department,  other  schemes 
were  also  used,  including  listings  by  time,  by  course 
number,  and  alphabetically,  by  name.  Version  A was 
intended  to  serve  as  a baseline  of  minimal  performance 
and  maximal  task  time.  Armchair  analysis  suggested 
that  organization  alphabetical  listings  by  course  title 
and  listing  by  course  number  was  unlikely  to  match 
anyone’s  procedural  or  declarative  knowledge  schema 
for  this  kind  of  information.  Thus,  it  was  used  for 
version  A. 

For  Version  B,  recall  and  interview  with  individuals 
who  had  attended  college  prior  to  the  computerization 
of  registration  led  to  organization  by  information  type 
(student  record,  departmental  course  listing, 
requirements,  etc.),  much  as  it  had  been  in  my  student 
days.  To  prevent  the  participants  in  Conditions  A and 
B from  reorganizing  the  information  by  opening 
multiple  windows,  the  metaphor  of  an  electronic  book 
was  used,  with  “pages”  for  each  kind  of  information 
above.  The  book  could  only  be  open  to  one  page  at  a 
time. 


Apparatus  Design  Methods:  Establishing  the  correct 
information  structure  can  be  an  iterative,  almost 
circular  process  as  procedures  can  change  when  tools 
change.  The  question  herein  is  not  how  to  design 
displays,  but  what  are  the  effects  of  information 
organization  on  performance.  Three  prototype 
information  presentation  schemes  were  developed. 
Versions  A and  B were  modeled  on  the  registration 
materials  used  by  many  pre-internet  generations  of 
students  These  included  student  record;  a catalogue 
listing  general  degree  requirements  and  specific 
requirements  for  each  major,  course  descriptions; 
prerequisites  for  each  course;  a schedule  of  courses  for 
the  upcoming  semester;  a student  schedule  sheet  for 
recording  selections;  and  an  informal  rating  of  faculty 
published  in  the  student  newspaper.  Students  had  to 
determine  the  availability  of  seats  for  each  course,  by 
going  to  each  department.  For  this  experiment,  this  last 
step  was  consolidated  into  a single  list.  Selecting 
courses  required  multi-way  comparisons  among  the 
sources  of  information,  and  across  many  pages  in  each. 
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Figure  1:  Steps  used  by  a typical  student  to  complete  a 
course  schedule. 

Version  C was  modeled  on  modem  computer 
registration.  This  was  designed,  not  by  attempting  to 
recreate  a historic  artifact,  but  by  a bootstrapping 
procedure  based  on  two  individuals  pilot  testing 
versions  A and  B.  Figure  1 shows  all  of  the  steps  used 
by  these  individuals1.  The  many  steps  used  different 
subsets  of  the  available  information  (see  Figure  1). 
These  subsets  were  used  sequentially,  as  a single 
cluster,  although  the  order  was  not  invariant.  For 
example,  two  clusters  were  used  to  make  initial  course 
selections.  These  were  (1)  requirements  and  student 
record  of  courses  already  taken,  and  (2)  course 
prerequisites,  student  record,  and  times  of  potential 
courses. 

The  same  information  could  be  used  by  several  steps. 
To  accommodate  all  possible  steps  in  this  procedural 
approach  to  information  organization,  the  clusters  of 
information  most  often  used  by  a single  step  were 
displayed  in  the  same  window  or  in  other  windows  that 


could  be  open  simultaneously.  Thus,  only  eye 
movements  were  necessary  to  obtain  all  the 
information  used  by  any  step. 

PHASE  2 BEHAVIORAL  EXPERIMENT 
Participants:  The  two  a priori  requirements  for 
participants  were  that  they  be  recent  college  students 
with  a minimum  of  eight  semesters  and  that  they  had 
registered  for  college  classes  within  the  past  five  years. 
They  could  be  considered  experienced  at  the  putative 
task.  There  were  10  women  and  26  men  evenly 
distributed  across  the  three  conditions.  They  had  last 
registered  for  college  courses  an  average  of  2.86  years 
prior  to  the  experiment.  Educational  level  of  the 
sample  ranged  from  bachelors  to  doctorate  degree.  The 
experiment  used  a between  subjects  design  with  twelve 
participants  per  group. 

Apparatus  and  materials:  The  task  was  a second  year 
college  registration.  Participants  were  given  goals, 
student  records  of  course  and  grade  history, 
prerequisites,  a campus  map  with  walking  times,  and 
instructor  ratings.  The  task  was  timed  and  courses 
closed,  dependent  on  the  elapsed  time.  All 
experimental  material  was  developed  in  Supercard  and 
presented  on  a Macintosh  computer  with  a 19  inch 
color  display.  Conditions  A and  B used  a booklet-like 
format  with  only  a single  page  visible  at  a time.  The 
page  numbers  in  the  Table  of  Contents  were  hypertext 
links  to  the  listed  information.  Categories  are  listed  in 
Table  2. 

Table  2:  Categories  of  information  in  Table  of 
Contents 

■ Goals 

■ Student  record 

■ General  requirements 

■ Departmental  requirements  for  Major 

■ Course  Prerequisites 

■ Courses  Schedule 

■ Campus  map 

■ Table  of  Instructor  ratings 

■ Table  of  class  size  and  seats  remaining 

In  condition  A,  all  information  in  the  course  schedule 
and  prerequisites  sections  was  listed  in  alphabetical 
order.  Thus,  Introduction  to  Psychology  followed 
Introduction  to  Physics.  As  courses  were  not  listed  by 
course  title  (only  course  number)  in  the  requirements 
and  student  record  sections,  this  format  required  a 
search  information  retrieval  strategy.  In  condition  B 
course  schedule  and  prerequisites  were  listed  by 
department,  and  sequentially  by  number  within 
department. 


1 Not  all  steps  were  used  on  every  trial. 
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Condition  C used  the  procedural  format  developed  in 
Phase  1.  It  used  a computer  registration  analogy  with 
access  to  information  via  a menu.  Multiple  movable 
windows  could  be  open  simultaneously.  These  were 
scrollable  and  resizable  when  required  (e.g.,  course 
schedule,  registration  card,  and  any  other  card  with 
more  than  about  10  lines  of  information).  As  the  screen 
and  text  were  of  the  same  size,  approximately  the  same 
quantity  of  information  was  visible  in  all  conditions. 

In  all  three  versions  of  the  task,  the  program  recorded 
windows  opening,  buttons  being  pushed,  typed  text, 
and  the  time  (in  ticks)  associated  with  each  interaction. 

Procedure:  The  experiment  took  place  in  a small, 
sound-damped,  experimental  room.  All  instructions 
were  presented  on  the  computer  screen.  Participants 
were  given  introductory  instruction  on  manipulation  of 
the  objects  used  in  the  program  and  on  the  task.  There 
was  a practice  task  for  each  condition  that  duplicated 
all  of  the  screens,  interactions,  and  information  types. 
During  the  practice  and  before  the  actual  experimental 
trial,  participants  were  invited  to  ask  questions, 
however,  questions  about  strategy  were  not  answered. 
No  questions  were  answered  after  the  experimental 
trial  began.  At  the  end  of  the  experimental  task, 
participants  were  asked  to  complete  a computerized 
questionnaire  and  were  debriefed.  The  questionnaire 
was  designed  to  ascertain  recall  of  relevant 
information,  task  strategy,  computer  and  college 
registration  expertise,  and  any  comments  about  the 
experiment. 

STATISTICAL  RESULTS 

Three  classes  of  dependent  measures  were  used.  The 
first  was  total  time-on-task  (T).  The  second  reflected 
outcome  performance  (P),  and  the  third  measured 
processing  activity  (A). 

Performance  was  defined  as  the  summation  of  the 
following  four  performance  measures:  the  number  of 
credits  successfully  registered  (P,),  the  sum  of  the 
requirements  scores  for  all  courses  registered  (P2),  the 
sum  of  the  scheduling  difficulty  scores  for  all  courses 
registered  (P3),  and  the  average  preference  score  for  the 
instructors  of  all  selected  courses  (P4).  Scheduling 
difficulty  was  computed  as  the  number  of  seats  in 
courses  that  would  satisfy  requirements  times  the 
number  of  credits,  weighted  by  the  scheduling 
priorities  given  in  the  goal  list.  Each  of  the  four  scores 
was  determined  a priori  and  was  reflected  in  the  goal 
set  given  to  the  participants. 

The  processing  measures  captured  various  aspects  of 
the  effort  participants  put  into  the  task.  These  included, 
number  of  registration  attempts  (A,),  number  of  class 
close-outs  (A2),  number  of  windows  used  (A,),  and 
number  of  times  the  subject  iterated  back  and  forth, 


between  any  two  windows  (A4).  One  full  cycle  from 
window  a to  b to  a to  b was  counted  as  one  iteration. 
These  measures  were  combined  into  an  overall 
efficiency  measure.  Efficiency,  E,  was  defined  as  the 
ratio  of  mean  overall  performance,  m^p^  to  mean 

amount  of  processing,  plus  total  time-on-task 

required  to  achieve  that  level  of  performance,  T: 

E = ny^m^+T)  (1) 

The  groups  did  not  differ  on  any  of  the  measures  of 
computer  experience  or  college  registration  experience. 
Table  3 shows  means  and  standard  deviations  for  all 
behavioral  measures  (T,  P,  A,  & E).  To  facilitate 
comparisons  among  measures,  all  scores  were 
transformed  into  standard  scores  with  a mean  of  50  and 
a standard  deviation  of  10. 

Time-on-task:  While  means  were  not  significantly 
different  among  groups,  variances  were  large,  time-on- 
task  did  contribute  to  individual  performance 
differences.  To  account  for  differences  in  time  taken  by 
individual  participants,  performance  and  processing 
measures  were  computed  per  unit  time.  While  not 
statistically  different  due  to  large  variances,  the  trend 
was  surprising.  Versions  A and  C appeared  equally  fast 
(Figure  2). 


Group 


Figure  2:  Mean  time-on-task  and  mean  performance. 

Performance  Measures:  Overall  performance, 

superimposed  over  time-on-task  shows  the  relationship 
between  the  two.  Those  using  version  A found  it  so 
difficult  that  they  basically  gave  up  trying  perform 
well.  They  just  wanted  to  complete  the  task  quickly. 
People  using  version  B found  that  they  could  complete 
the  task,  but  it  took  considerable  time  and  effort. 

Decomposing  the  performance  measures  provides  the 
supporting  detail  necessary  to  understand  this  effect. 
The  performance  measures  appeared  to  be  composed 
of  two  compound  measures  that  behaved  very 
differently  from  one  another  (see  Figure  3).  The  first, 
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P,  2,  was  composed  of  the  more  concrete  performance 
measures;  P,,  the  number  of  credits  successfully 
registered  and  P2,  the  sum  of  the  requirement 
satisfaction  scores  for  all  registered  courses.  The  tasks 
represented  by  these  measures  were  essential  for 
completion  of  the  course  schedule  and  did  not  reflect 
differences  in  performance.  They  replicated  minimal  or 
baseline  performance.  Although  there  appears  to  be  a 
slight  trend  toward  better  performance  for  groups  B 
and  C,  this  was  not  significant,  F<1 .0. 

54 


Group 

Figure  3:  Mean  performance  on  two  compound 
measures  for  each  display  format. 

Processing  Measures:  The  process  measures  also 
showed  two  patterns  of  behavior  (see  Figure  4).  For 


Table  3:  Means  and  standard  deviations  for  all 


measures  for  all  groups. 


Alphabetical 
M SD 

Functional 
M SD 

Procedural 
M SD 

Time  on 

task 

49.24 

9.76 

52.54 

10.62 

48.21 

9.94 

Performance  Measures 

P, 

48.40 

11.77 

50.97 

6.36 

50.60 

11.62 

P3 

48.33 

12.20 

50.17 

8.87 

51.52 

9.23 

P3 

49.65 

10.07 

47.69 

11.35 

52.59 

8.73 

P4 

49.67 

11.46 

46.61 

8.60 

53.82 

9.29 

Process  Measures 

A, 

48.89 

5.43 

53.47 

15.04 

47.21 

5.76 

A, 

51.01 

12.33 

52.15 

10.93 

46.44 

3.95 

A, 

55.78 

8.98 

54.21 

7.60 

40.02 

3.92 

a4 

56.65 

9.65 

53.84 

6.37 

39.51 

0.42 

Efficiency  Ratio 

ease  of  analyses  anSPdiscussira,  Siese  shafts  c°aflfd 
A,  2 and  A3  4,  with  the  understanding  that  the  two 
components  of  each  compound  measure  displayed  the 
same  pattern  of  results.  The  first  compound  measure 
was  composed  of  measures  A,;  registration  attempts 
and  A2,  number  of  close-outs.  These  are  both 
indications  of  difficulties  with  the  task,  rather  than  the 
information  format.  There  were  no  significant 


differences  among  the  conditions  on  this  compound 
measure,  F(2,33)  = 1.71,  n.s. 


Group 

Figure  4:  Mean  processing  score  on  two  compound 
measures  for  each  display  format. 

The  second  compound  performance  measure,  P34, 
was  composed  of  more  difficult,  evaluative  and 
integrative  tasks;  P3,  the  sum  of  the  scheduling 
difficulty  scores  for  all  courses  registered,  and  P4,  the 
average  preference  score  for  the  instructors  of  all 
selected  courses  (see  again  Figure  3).  These  measures 
appeared  to  reflect  an  added  effort  to  perform  well, 
when  possible.  P3(4  showed  a paradoxical  dip  in 
performance  with  the  functional  display  organization, 
B.  This  quadratic  trend  was  marginally  significant, 
F(l,33)=4.07,  £=.05. 

The  second  pair  of  processing  measures;  A,,  number  of 
windows  and  A4,  number  of  iterations,  are  related  to 
information  accessibility  and  congruence  with 
procedural  needs.  If  the  information  format  does  not 
match  the  sequences  used  by  procedural  knowledge, 
the  individual  must  collect  it  from  where  it  is 
(indicated  by  A3)  and  then  create  the  sequence  in 
working  memory  (encoding  and  sequencing  indicated 
by  A4).  There  were  significant  differences  among  the 
groups  on  this  pair  of  measures,  F(2,33)  = 43.04, 
£<.001.  This  was  a very  robust  effect  with  q2  = 0.72 
and,  in  a post  hoc  test  for  trend,  the  quadratic  trend  was 
significant,  F(l,33)  = 13.104,  £<.005. 

Efficiency:  The  processing,  performance  and  time-on- 
task  measures  combined  to  evaluate  the  effect  of 
information  organization  of  decision  making 
efficiency.  Participants  who  used  the  procedural 
information  format  were  significantly  more  efficient  in 
their  decision  making  than  were  those  using  either  of 
the  other  two  formats,  F(2,33)=7.29,  £<.005  (see 
Figure  5).  This  was  a robust  effect,  with  r)2  = 0.32.  In  a 
post  hoc  test  for  trend,  the  quadratic  trend  was 
significant,  F(l,33)  = 4.32,  p<.05. 
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Figure  5:  Efficiency  (performance  divided  by 

processing  effort  and  time)  for  each  of  the  three  display 
formats. 

PROCEDURAL  ANALYSIS 

The  first  task  of  the  procedural  analysis  was  to  evaluate 
the  relationship  between  overall  performance  and 
overall  procedural  processes.  This  relationship  can  best 
be  understood  by  examining  Figure  6.  As  can  be  seen, 
there  was  an  inverse  relationship  among  these 
measures.  Processing  variables  were  moderately 
predictive  of  total  performance  score,  R2=  0.42. 


Figure  6:  The  relationship  between  performance  and 
process  measures  for  each  format. 

Iterations:  The  iteration  measure  is  a reflection  of  the 
efficiency  of  procedures  used  by  participants  under 
each  of  the  three  conditions.  In  conditions  A and  B, 
participants  physically  iterated  between  pairs  of 
information  while  in  condition  C they  typically 
positioned  information  windows  so  that  information 
used  in  the  current  step  could  be  accessed  with  eye 
movements  and  did  not  require  mouse  actions.  A 
typical  pattern  for  the  participants  in  condition  A was 
to  iterate  between  the  schedule  page  and  virtually 
every  other  page.  However,  there  were  numerous 
iterations  among  other  pages.  For  participants  in 
condition  C,  virtually  all  of  the  iterations  were  between 
the  schedule  page  and  one  of  the  pages  listing  course 


schedules.  There  were  very  few  iterations  for 
participants  in  condition  C and  these  were  different  for 
each  subject.  Figure  7 shows  typical  patterns  of 
iterations. 


Figure  7:  Typical  patterns  of  iterations  for  participants 
in  each  of  the  three  conditions.  Node  #1  is  Table  of 
Contents. 


Sequences:  Just  as  the  number  and  pattern  of  iterations 
differed  by  condition,  other  aspects  of  performance 
differed  by  condition.  Unfortunately,  neither  verbal 
protocols  nor  eye  gaze  data  were  collected  so  one  can 
only  infer  goals.  Self  reports  in  the  follow-up 
questionnaire  shed  no  light  on  the  question  because 
there  were  no  differences  among  groups  and  because 
participants  often  reported  placing  no  weight  on  a 
source  of  information  that  they  examined  frequently  or, 
conversely,  placing  heavy  weight  on  information  that 
they  never  accessed. 

Sequences  of  pages  (Conditions  A and  B)  or  windows 
(Condition  C)  provide  insight  into  goals  and  validation 
of  the  procedure  described  in  Figure  1 . Several  of  the 
sequences  were  found  in  all  conditions  including  those 
used  for  procedures  1 and  2.  Only  participants  in 
Condition  C were  able  to  see  a listing  of  courses  by 
time  slot.  The  majority  of  them  used  this  display,  but 
only  for  the  last  one  or  two  course  selections. 

The  sequence  of  the  requirements  listings  either 
preceded  or  followed  by  student  record  (procedural 
step  1,  Figure  1)  is  used  to  illustrate  differences  among 
the  groups.  Groups  A and  B examined  requirements  an 
average  of  5.85  (SD  = 1.29)  and  6.75  (SD  = 0.94) 
times,  respectively  while  group  C reviewed  this 
important  information  an  average  of  10.75  (SD  = 1 .01) 
times.  Moreover,  review  of  this  pair  of  windows  was 
not  evenly  distributed  across  the  duration  of  the  task.  It 
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appeared  more  frequently  prior  to  early  course 
selections.  Apparently  as  the  possibilities  narrowed, 
many  participants  chose  to  skip  step  1 in  the  procedure. 

DISCUSSION 

Providing  guidance  for  the  development  of  interfaces 
that  support  efficient  (proficient  and  timely)  decision 
making  was  a major  motivator  for  this  study.  The  most 
obvious  conclusion  is  that  these  results  indicate  the 
importance  of  information  organization  that  is 
congruent  with  procedural  knowledge.  Moreover,  they 
show  the  impact  of  such  organizational  schemes  on 
efficient  performance.  A more  detailed  examination  of 
the  data  indicates  that  differences  in  support  for 
procedural  knowledge  differentially  change  task 
procedures.  Evidence  for  these  differences  are 
provided  by  all  three  sets  of  results  reported  here, 
performance  score  components,  processing  activity 
components,  and  the  iterations  picture. 

Performance:  Differences  in  the  two  compound 
performance  scores,  P, ,,  (the  more  concrete,  required 
performance  measures)  and  P34,  (the  optional 
evaluative  and  integrative  measures)  suggest  that 
motivation  plays  a subtle  role  in  the  equation. 
Performance  on  the  Pl2  measure  was  flat,  but  with 
large  individual  differences.  A ceiling  effect  may  have 
contributed  to  the  lack  of  systematic  differences. 

P34  performance  was  clearly  affected  by  some  aspect 
of  the  tool.  As  these  measures  related  to  qualitative 
performance,  they  might  reflect  the  ease  of  use  for  the 
different  organizational  schemes.  Clearly,  Version  C 
provided  the  cognitive  affordances  for  better 
performance.  The  processing  scores  (see  below) 
contribute  to  this  conclusion. 

Surprisingly,  performance  was  not  poorer  on  any 
measure  for  Condition  A.  Although  that  version  of  the 
task  was  intended  to  provide  the  least  support  for 
procedural  knowledge,  it  did  not  differ  from  the 
traditional,  department  (like-with-like)  organizational 
scheme.  Might  it  be  true  that  any  information 
organization  that  is  not  congruent  with  procedural 
knowledge  restricts  performance? 

Processing  Activity:  As  with  the  performance 

compound  measures,  differences  in  the  two  compound 
processing  measures  showed  different  patterns. 
Compound  measure  A,,  reflects  the  task  difficulty  and, 
not  surprisingly,  did  not  differ  among  the  three 
conditions.  This  lack  of  difference  verifies  that  the  task 
could  be  accomplished  with  any  of  the  three  versions. 

Compound  score  A34  measures  physical  interactions 
and  reflects  congruence  between  procedural  knowledge 
and  affordances  in  the  tool.  The  systematic  differences 
in  Aj4  indicate  that  the  task  was  more  easily  and 
efficiently  accomplished  with  version  C.  Again,  there 
were  no  differences  between  versions  A and  B. 


CONCLUSIONS  AND  IMPLICATIONS  FOR 
BATTLESPACE  MANAGEMENT  SYSTEMS 

We  have  seen  that  an  information  organization  scheme 
based  on  procedural  knowledge  (Condition  C)  can 
facilitate  performance  at  a complex,  time-driven  task. 
Moreover,  performance  without  a reasoning-congruent 
information  scheme  hindered  performance,  regardless 
of  the  information  organization.  When  the  decision 
maker  must  expend  both  time  and  cognitive  resources 
to  compensate  for  the  tool,  those  resources  are  not 
available  to  perform  the  task.  Thus,  the  right 
organization  scheme  can  provide  the  affordances  for 
improved  cognitive  performance. 

How  would  this  work  in  a Battlespace  Management 
System?  As  I am  most  familiar  with  submarine  systems 
and  with  meteorological  systems  I’ll  use  one  of  those, 
the  submarine,  as  an  example.  The  submarine  systems 
include  multiple  sensor  performance  prediction 
algorithms,  sonar  sensors,  target-motion-analysis 
algorithms,  and  battlespace  displays.  These  tools 
correspond  to  the  functions  of  search,  detect,  track, 
classify,  localize,  etc.  However,  when  we  examine  the 
behavior  of  expert  submariners,  they  do  not  limit 
themselves  to  this  sequence  (Gray,  Kirschenbaum,  & 
Ehret,  1997;  Kirschenbaum,  1992).  They  iterate  among 
the  tools  as  they  employ  specific  information  gathering 
strategies.  Thus  a Battlespace  Management  System  for 
submariners  might,  for  example,  facilitate  comparing 
the  output  of  a target-motion-analysis  tool  to  the  sonar 
traces  at  that  bearing.  Would  the  proposed  course, 
speed,  and  range  actually  fall  within  the  sonar  trace,  as 
displayed?  How  well  would  it  match  the  region  where 
the  sonar  could  detect?  We  are  currently  building 
displays  to  answer  these  questions  by  showing  these 
detection  regions  in  3-D,  along  with  the  possible 
tracks.  Thus,  we  facilitate  the  very  comparison 
procedures  that  we  have  observed  submarine  decision 
makers  using. 

The  submarine  3-D  display  work  is  just  beginning.  The 
effects  on  performance  have  yet  to  be  tested.  The 
approach  is  much  like  that  used  in  the  experiment 
reported  above.  If  the  results  replicate  in  this  domain 
differences  in  affordances  for  procedural  knowledge 
will  again  support  differences  in  performance.  While 
information  systems  have  always  been  developed  by 
analyzing  perceived  needs,  a radically  different 
suggestion  is  to  design  information  management 
decision  aids  the  way  new  walkways  are  sometimes 
planned.  That  is,  where  natural  paths  occur  because  of 
repeated  use  by  pedestrians,  constructed  (concrete, 
macadam,  gravel,  etc.)  pathways  are  built.  These  are 
not  planned  a priori,  but  develop  from  use.  Only  after 
the  grass  has  been  worn,  are  constructed  paths  built.  In 
the  same  way,  the  information  organization  schemes 
should  provide  affordances  for  knowledge  in  the  head  - 
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- the  procedural  paths  we  create  through  the  task  and 
the  information. 
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